» Optuna Params Tuning Report
AutoML Leaderboard
| Best model | name | model_type | metric_type | metric_value | train_time |
|---|---|---|---|---|---|
| 1_Optuna_LightGBM | LightGBM | logloss | 0.267532 | 19.48 | |
| 2_Optuna_Xgboost | Xgboost | logloss | 0.271852 | 17.06 | |
| 3_Optuna_RandomForest | Random Forest | logloss | 0.273661 | 17.25 | |
| the best | Ensemble | Ensemble | logloss | 0.264678 | 0.91 |
AutoML Performance
AutoML Performance Boxplot
Features Importance
Spearman Correlation of Models
Summary of Ensemble
Ensemble structure
| Model | Weight |
|---|---|
| 1_Optuna_LightGBM | 1 |
| 2_Optuna_Xgboost | 1 |
| 3_Optuna_RandomForest | 1 |
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.264678 | nan |
| auc | 0.954611 | nan |
| f1 | 0.888056 | 0.45137 |
| accuracy | 0.8955 | 0.45137 |
| precision | 1 | 0.980023 |
| recall | 1 | 0.00364726 |
| mcc | 0.790857 | 0.495641 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.264678 | nan |
| auc | 0.954611 | nan |
| f1 | 0.888056 | 0.45137 |
| accuracy | 0.8955 | 0.45137 |
| precision | 0.902067 | 0.45137 |
| recall | 0.874473 | 0.45137 |
| mcc | 0.790451 | 0.45137 |
Confusion matrix (at threshold=0.45137)
| Predicted as Çerçevelik | Predicted as Ürgüp Sivrisi | |
|---|---|---|
| Labeled as Çerçevelik | 962 | 90 |
| Labeled as Ürgüp Sivrisi | 119 | 829 |
Learning curves
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 3_Optuna_RandomForest
Random Forest
- n_jobs: -1
- criterion: entropy
- max_features: 0.036068463600684
- min_samples_split: 19
- max_depth: 28
- eval_metric_name: logloss
- min_samples_leaf: 6
- max_steps: 10
- explain_level: 2
Validation
- validation_type: kfold
- k_folds: 10
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
16.7 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.273661 | nan |
| auc | 0.95145 | nan |
| f1 | 0.885003 | 0.445593 |
| accuracy | 0.8925 | 0.476828 |
| precision | 1 | 0.998986 |
| recall | 1 | 0 |
| mcc | 0.785535 | 0.541451 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.273661 | nan |
| auc | 0.95145 | nan |
| f1 | 0.884842 | 0.476828 |
| accuracy | 0.8925 | 0.476828 |
| precision | 0.898803 | 0.476828 |
| recall | 0.871308 | 0.476828 |
| mcc | 0.784423 | 0.476828 |
Confusion matrix (at threshold=0.476828)
| Predicted as Çerçevelik | Predicted as Ürgüp Sivrisi | |
|---|---|---|
| Labeled as Çerçevelik | 959 | 93 |
| Labeled as Ürgüp Sivrisi | 122 | 826 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 1_Optuna_LightGBM
LightGBM
- n_jobs: -1
- objective: binary
- num_leaves: 1553
- learning_rate: 0.1
- feature_fraction: 0.6372942274996019
- bagging_fraction: 0.649963737714838
- min_data_in_leaf: 2
- metric: binary_logloss
- custom_eval_metric_name: None
- lambda_l1: 0.00015613577890072756
- lambda_l2: 0.005003670155001035
- bagging_freq: 5
- extra_trees: True
- num_boost_round: 1000
- early_stopping_rounds: 50
- cat_feature: []
- feature_pre_filter: False
- explain_level: 2
Validation
- validation_type: kfold
- k_folds: 10
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
18.8 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.267532 | nan |
| auc | 0.953404 | nan |
| f1 | 0.885605 | 0.543914 |
| accuracy | 0.8955 | 0.543914 |
| precision | 1 | 0.982488 |
| recall | 1 | 0.00240858 |
| mcc | 0.791587 | 0.543914 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.267532 | nan |
| auc | 0.953404 | nan |
| f1 | 0.885605 | 0.543914 |
| accuracy | 0.8955 | 0.543914 |
| precision | 0.920364 | 0.543914 |
| recall | 0.853376 | 0.543914 |
| mcc | 0.791587 | 0.543914 |
Confusion matrix (at threshold=0.543914)
| Predicted as Çerçevelik | Predicted as Ürgüp Sivrisi | |
|---|---|---|
| Labeled as Çerçevelik | 982 | 70 |
| Labeled as Ürgüp Sivrisi | 139 | 809 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
SHAP Importance
SHAP Dependence plots
Dependence (Fold 1)
Dependence (Fold 2)
Dependence (Fold 3)
Dependence (Fold 4)
Dependence (Fold 5)
Dependence (Fold 6)
Dependence (Fold 7)
Dependence (Fold 8)
Dependence (Fold 9)
Dependence (Fold 10)
SHAP Decision plots
Top-10 Worst decisions for class 0 (Fold 1)
Top-10 Worst decisions for class 0 (Fold 2)
Top-10 Worst decisions for class 0 (Fold 3)
Top-10 Worst decisions for class 0 (Fold 4)
Top-10 Worst decisions for class 0 (Fold 5)
Top-10 Worst decisions for class 0 (Fold 6)
Top-10 Worst decisions for class 0 (Fold 7)
Top-10 Worst decisions for class 0 (Fold 8)
Top-10 Worst decisions for class 0 (Fold 9)
Top-10 Worst decisions for class 0 (Fold 10)
Top-10 Best decisions for class 0 (Fold 1)
Top-10 Best decisions for class 0 (Fold 2)
Top-10 Best decisions for class 0 (Fold 3)
Top-10 Best decisions for class 0 (Fold 4)
Top-10 Best decisions for class 0 (Fold 5)
Top-10 Best decisions for class 0 (Fold 6)
Top-10 Best decisions for class 0 (Fold 7)
Top-10 Best decisions for class 0 (Fold 8)
Top-10 Best decisions for class 0 (Fold 9)
Top-10 Best decisions for class 0 (Fold 10)
Top-10 Worst decisions for class 1 (Fold 1)
Top-10 Worst decisions for class 1 (Fold 2)
Top-10 Worst decisions for class 1 (Fold 3)
Top-10 Worst decisions for class 1 (Fold 4)
Top-10 Worst decisions for class 1 (Fold 5)
Top-10 Worst decisions for class 1 (Fold 6)
Top-10 Worst decisions for class 1 (Fold 7)
Top-10 Worst decisions for class 1 (Fold 8)
Top-10 Worst decisions for class 1 (Fold 9)
Top-10 Worst decisions for class 1 (Fold 10)
Top-10 Best decisions for class 1 (Fold 1)
Top-10 Best decisions for class 1 (Fold 2)
Top-10 Best decisions for class 1 (Fold 3)
Top-10 Best decisions for class 1 (Fold 4)
Top-10 Best decisions for class 1 (Fold 5)
Top-10 Best decisions for class 1 (Fold 6)
Top-10 Best decisions for class 1 (Fold 7)
Top-10 Best decisions for class 1 (Fold 8)
Top-10 Best decisions for class 1 (Fold 9)
Top-10 Best decisions for class 1 (Fold 10)
Summary of 2_Optuna_Xgboost
Extreme Gradient Boosting (Xgboost)
- n_jobs: -1
- objective: binary:logistic
- eta: 0.05
- max_depth: 5
- min_child_weight: 1
- subsample: 0.5758258292131195
- colsample_bytree: 0.3228568309919116
- eval_metric: logloss
- lambda: 1.4668648312244355e-06
- alpha: 8.22322663436197e-07
- max_rounds: 1000
- early_stopping_rounds: 50
- explain_level: 2
Validation
- validation_type: kfold
- k_folds: 10
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
16.4 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.271852 | nan |
| auc | 0.952193 | nan |
| f1 | 0.884842 | 0.442869 |
| accuracy | 0.8925 | 0.442869 |
| precision | 1 | 0.991096 |
| recall | 1 | 0.00262342 |
| mcc | 0.784423 | 0.442869 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.271852 | nan |
| auc | 0.952193 | nan |
| f1 | 0.884842 | 0.442869 |
| accuracy | 0.8925 | 0.442869 |
| precision | 0.898803 | 0.442869 |
| recall | 0.871308 | 0.442869 |
| mcc | 0.784423 | 0.442869 |
Confusion matrix (at threshold=0.442869)
| Predicted as Çerçevelik | Predicted as Ürgüp Sivrisi | |
|---|---|---|
| Labeled as Çerçevelik | 959 | 93 |
| Labeled as Ürgüp Sivrisi | 122 | 826 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
SHAP Importance
SHAP Dependence plots
Dependence (Fold 1)
Dependence (Fold 2)
Dependence (Fold 3)
Dependence (Fold 4)
Dependence (Fold 5)
Dependence (Fold 6)
Dependence (Fold 7)
Dependence (Fold 8)
Dependence (Fold 9)
Dependence (Fold 10)
SHAP Decision plots
Top-10 Worst decisions for class 0 (Fold 1)
Top-10 Worst decisions for class 0 (Fold 2)
Top-10 Worst decisions for class 0 (Fold 3)
Top-10 Worst decisions for class 0 (Fold 4)
Top-10 Worst decisions for class 0 (Fold 5)
Top-10 Worst decisions for class 0 (Fold 6)
Top-10 Worst decisions for class 0 (Fold 7)
Top-10 Worst decisions for class 0 (Fold 8)
Top-10 Worst decisions for class 0 (Fold 9)
Top-10 Worst decisions for class 0 (Fold 10)
Top-10 Best decisions for class 0 (Fold 1)
Top-10 Best decisions for class 0 (Fold 2)
Top-10 Best decisions for class 0 (Fold 3)
Top-10 Best decisions for class 0 (Fold 4)
Top-10 Best decisions for class 0 (Fold 5)
Top-10 Best decisions for class 0 (Fold 6)
Top-10 Best decisions for class 0 (Fold 7)
Top-10 Best decisions for class 0 (Fold 8)
Top-10 Best decisions for class 0 (Fold 9)
Top-10 Best decisions for class 0 (Fold 10)
Top-10 Worst decisions for class 1 (Fold 1)
Top-10 Worst decisions for class 1 (Fold 2)
Top-10 Worst decisions for class 1 (Fold 3)
Top-10 Worst decisions for class 1 (Fold 4)
Top-10 Worst decisions for class 1 (Fold 5)
Top-10 Worst decisions for class 1 (Fold 6)
Top-10 Worst decisions for class 1 (Fold 7)
Top-10 Worst decisions for class 1 (Fold 8)
Top-10 Worst decisions for class 1 (Fold 9)
Top-10 Worst decisions for class 1 (Fold 10)
Top-10 Best decisions for class 1 (Fold 1)
Top-10 Best decisions for class 1 (Fold 2)
Top-10 Best decisions for class 1 (Fold 3)
Top-10 Best decisions for class 1 (Fold 4)
Top-10 Best decisions for class 1 (Fold 5)
Top-10 Best decisions for class 1 (Fold 6)
Top-10 Best decisions for class 1 (Fold 7)
Top-10 Best decisions for class 1 (Fold 8)
Top-10 Best decisions for class 1 (Fold 9)
Top-10 Best decisions for class 1 (Fold 10)
Optuna tuning for LightGBM on original data
LightGBM Optimization History
LightGBM Parallel Coordinate
LightGBM Param Importances
Optuna tuning for Xgboost on original data
Xgboost Optimization History
Xgboost Parallel Coordinate
Xgboost Param Importances
Optuna tuning for Random Forest on original data
Random Forest Optimization History
Random Forest Parallel Coordinate
Random Forest Param Importances
